Chapter 25 : Multimodal interaction modeling

Virtual Environments (VE) represent advanced human computer interaction systems, communicating over several channels of information. The basic terms we need to describe human VE interaction are “communication channel” and “multimodality”. The communication channel is as a pathway between the user and the simulation that mediates the interaction. Multimodality is the quality of a system, which allows more that one modality (channel) to be used during human computer interaction. In a multimodal system the user communicates with computers via several modalities such as voice, gesture, gaze, visual, auditory, haptic, etc. Multimodal interaction modeling studies the mechanisms of integrating modalities in order to enhance human computer interaction. The popular view of Virtual Reality (VR) emphasizes the sensorial experience provided through high quality feedback devices, neglecting the user input component. In reality, a typical platform for developing VEs has both a computer providing visual and auditory feedback plus an input device controlled by the user. Most of the information is received through the visual channel, similar to the real world. The need for increased immersion and interaction motivates the designers to explore the integration of additional modalities and to take advantage of cross-modal effects. Adding several communication channels comes at the price of system complexity, cost, and of integration/synchronization problems. The benefits of multimodal input (more efficient human machine dialogue, increased naturalness of interaction, etc.) were demonstrated for recent multimedia systems. As multimedia evolves from keyboard and mouse input towards speech-gesture interfaces, VR systems should consider using multimodal input for interaction performance enhancement. While in multimedia systems the feedback to the user, as well as informational content, is limited to images and sound, VEs are candidates for complex multimodal human-computer interfaces. The next section in this chapter presents a human computer interaction model, with its I/O channels and their characteristics. Section 3 is dedicated to multimodal perception and cross-modal effects: sensorial transposition and sensorial redundancy. Section 4 presents design concepts for multimodal input modeling. VE architectures including multisensory feedback and multimodal input are reviewed in section 5. Section 6 summarizes the chapter.